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DETAILED ACTION 



1 . Claims 43-44 is canceled and claim 1 is amended on amendment filed on 
02/05/03. Claims 1-42 are pending in this application. 

Response to Arguments 

2. Applicant's arguments with respect to claims 1 , 10, 18, 25 and 32 have been 
considered but are moot in view of the new ground(s) of rejection. 
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Claim Rejections - 35 USC § 103 



The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

3. Claims 1,6-11,13-17 and 32-42 are rejected under 35 U.S.C. 1 03(a) as being 
unpatentable over Morag (US. Patent No. 6,324,545) in view of Tada et al. (US. 
Patent No. 5,572,728). 

Regarding on claims 1 and 9, Morag teaches a method comprising: 
monitoring feedback from the user as to which of the first and second multimedia 
objects are relevant to the search query (col. 6, lines 53-55); and annotating one or 
more of the multimedia objects, which are deemed relevant by the user, with keyword 
(col. 9, lines 35-40). 

Morag teaches downloading the acquired images to a personal computer and 
with software for manipulating the acquired images (col. 5, lines 57-59). Morag does 
not explicitly teach identifying, in response to a search query, first multimedia objects 
having an associated keyword that matches a keyword in the search query and second 
multimedia objects that have content features similar to those of the first multimedia 
objects; and presenting the first and second multimedia objects to a user. However, 
Tada teaches, "images retrieved by a retrieval keyword are displayed in the form of a 
rectangular parallelepiped for each relevant item" (col. 10, lines 9-11). In addition, Tada 
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also teaches, "the image volume 22 corresponding to the retrieval keyword, and is 
generated by collecting related image" (col. 11, lines 21-23). This teaches the second 
set of images is collected. Therefore, it would have been obvious to one ordinary skill in 
the art at the time of the invention was made to combine the teaching of Morag and 
Tada because retrieving the images by the keyword and also at the same time 
retrieving the related images would allow the search to be more thoughtful. 

Regarding on claim 6, Morag teaches training how the first and second 
multimedia objects are identified based on the user's feedback (col. 9, lines 35-40). 

Regarding on claim 7, Tada teaches refining the search to identify additional 
multimedia objects that contain content features similar to those of the multimedia 
objects indicated by the user as being relevant. 

Regarding on claim 8, Morag teaches the multimedia objects comprise one of 
digital images, video objects, and audio objects (col. 5, line 50). 

Regarding on claims 10 and 17, Morag teaches a method comprising: 
monitoring feedback from a user to whether the multimedia objects are relevant 
to a keyword in a search query (col. 6, lines 53-55); and annotating the multimedia 
objects based on the user's feedback (col. 9, lines 35-4). Morag does not explicitly 
teach iteratively retrieving multimedia objects from a database. However, Tada 
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teaches, "setting the new retrieval conditions by use information other than the 
displayed images, and by executing re-retrieval" (col. 10, lines 21-23). Since the claim 
is not explicit, therefore, the re-retrieval would be able to read into the claim. Therefore, 
it would have been obvious to one ordinary skill in the art at the time of the invention 
was made to combine the teaching of Tada and Morag because utilizing the re-retrieval 
process would allow other results to be retrieved when the first one does not retrieve the 
right one. 

Regarding on claim 11, Tada teaches the retrieving comprises using content- 
based information retrieval to retrieve the multimedia objects (col. 4, lines 60-65). 

Regarding on claim 13, Morag teaches the monitoring comprises monitoring both 
feature-based relevance feedback and semantic-based relevance feedback (col. 9, lines 
35-40). 

Regarding on claim 14, Morag teaches the annotating is hidden from the user 
(col. 9, lines 35-40). 

Regarding on claim 15, Morag teaches the annotating comprises: 

in an event that a particular multimedia object is deemed relevant by the user 

and is not yet annotated with the keyword, adding the keyword to the particular 

multimedia object (col. 9, lines 35-40); and 
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in an event that the particular multimedia object is deemed relevant by the user 
and is already annotated with the keyword, weakening an association between the 
keyword and the particular multimedia object (col. 9, lines 35-40) 

Regarding on claim 16, Morag teaches the annotating comprises: 

In an event that a particular multimedia object is deemed irrelevant by the user 

and is already annotated with the keyword, weakening an association between the 

keyword and the particular multimedia object (col. 4, lines 48-52). 
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Regarding on claim 32, Morag teaches a system comprising: 
relevance feedback unit to capture a user's feedback as to whether the 
multimedia objects are relevant to the search query (col. 6, lines 53-55); and 
an annotation unit to annotate the multimedia objects based on the user's feedback (col. 
9, lines 35-40). 

Morag does not explicitly teach an information retrieval unit to retrieve multimedia 
objects from a database based on a search query. Tada teaches, "images retrieved by 
a retrieval keyword are displayed in the form of a rectangular parallelepiped for each 
relevant item. That is, when an image and a keyword related to an item are newly 
designated, images related to the keyword are automatically fetch from the data storing 
means 4" (col. 10, lines 9-14). This teaches the images are retrieved by keywords. 
Therefore, it would have been obvious to one ordinary skill in the art at the time of the 
invention was made to combine the teaching Tada and Morag because utilizing the 
keywords to search for the multi media objects would allow the images to be retrieved. 

Regarding on claim 33, Tada teaches the search query comprises a keyword- 
based search query having one or more keywords (col. 10, lines 9-14). 

Regarding on claim 34, Tada teaches the search query comprises a content- 
based search query having one or more content features (col. 10, lines 9-14). 
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Regarding on claim 35, Tada teaches the information retrieval unit employs both 
content-based information and semantic based information retrieval (col. 10, lines 9-14). 

Regarding on claim 36, Tada teaches the information retrieval unit comprises: 
A query handler to handle both keyword-based queries haying one or more 

search keywords and content-based queries having one or more content features of a 

multimedia object (col. 10, lines 9-14); and 

A feature and semantic matcher to identify at least one of (1 ) first multimedia 

object having keywords that match the search keywords from a keyword-based query, 

and (2) second multimedia objects having content features similar to the content 

features of a content-based query (col. 10, lines 9-14). 

Regarding on claim 37, Morag teaches the relevance feedback unit employs both 
feature-based relevance feedback and semantic-based relevance feedback (col. 9, lines 
35-40). 

Regarding on claim 38, Morag teaches the search query comprise a keyword- 
based search query having at least one keyword; and in an event that a particular 
multimedia object is deemed relevant by the user and is not yet annotated with 
keyword, the annotation units adds the keyword to the particular multimedia object (col. 
9, lines 35-40). 
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Regarding on claim 39, Morag teaches the search query comprises a keyword- 
based search query having at least one keyword; and in an event that a particular 
multimedia object is deemed relevant by the user and is already annotated with the 
keyword, the annotation unit strengthens an association between the keyword and the 
particular multimedia object (col. 9, lines 35-40). 

Regarding on claim 40, Morag teaches the search query comprises a keyword- 
based search query having at least one keyword; and in an event that a particular 
multimedia object is deemed irrelevant by the user and is already annotated with the 
keyword, weakening an association between the keyword and the particular multimedia 
object (col. 9, lines 35-40). 

Regarding on claim 41, Morag teaches the search query comprises a keyword- 
based search query having at least one keyword; and in an event that a particular 
multimedia object is deemed irrelevant by the user and is already annotated with the 
keyword, removing the keyword from the particular multimedia object (col. 9, lines 35- 
40). 

Regarding on claim 42, Morag teaches the relevance feedback unit comprises a 
feedback analyzer to train the system based on the user's feedback (col. 9, lines 35-40). 
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4. Claim 12 is rejected under 35 U.S.C. 103(a) as being unpatentable over Morag 
(US. Patent No. 6,324,545) in view of Tada etal. (US. Patent No. 5,572,728) and 
further in view of Barber et al. (US. Patent No. 5,579,471). 

Regarding on claim 12, Both Tada and Morag do not explicitly teach the 
retrieving comprises using both content-based information retrieval and semantic-based 
information retrieval to retrieve the multimedia objects. However, Barber teaches, "a 
computer display interface for constructing an image query used to access images in a 
database based on content includes a first selection window for specifying image color 
and including a plurality of color thumbnails, a third selection window for specifying 
image shapes... a fourth selection window including a plurality of category thumbnails 
which may denote keywords, text, or conditions on alpha-numeric data associated with 
an image" (col. 3, lines 23-33). This teaches the searching method utilizing both 
keywords and feature. Therefore, it would have been obvious to one ordinary skill in the 
art at the time of the invention was made to combine the teaching of Barber and Tada 
and Morag because utilizing both keyword and feature would allow the user to conduct 
a search more specific and retrieve the results as requested. 
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5. Claims 2-6 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Morag(US. Patent No. 6,324,545) in view of Tada et al. (US. Patent No. 5,572,728) 
and further in view of De Bonet (US. Patent No. 5,899,999). 

Regarding on claim 2, Morag and Tada teaches maintaining associations 
between the keywords and the multimedia objects, the associations being weighted to 
indicate how relevant the keywords are to the multimedia objects except for adjusting 
the weights of the associations based on the user's feedback. De Bonet teaches, "to 
retrieve an image, the system compares the vector for the test image, modified by the 
weights provided by the user, to the vector for each image in the database" (col. 4, lines 
38-41 ). This teaches adjusting the weight. Therefore, it would have been obvious to 
one ordinary skill in the art at the time of the invention was made to combine the 
teaching Morag and Tada and De Bonet because adjusting the weight to the images 
based on the user would allow the images to be more relevant to the keywords. 

Regarding on claim 3, De Bonet teaches the adjusting comprises increasing a 
weight of an association between the keyword and a particular multimedia object that is 
deemed relevant by the user (col. 4, lines 38-41 ). 
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Regarding on claim 4, De Bonet teaches the adjusting comprises decreasing a 
weight of an association between the keyword and a particular multimedia object that is 
deemed irrelevant by the user (col. 4, lines 34-41). 

Regarding on claim 5, De Bonet teaches removing the keyword from the 
particular multimedia object in an event that the weight is less than a threshold value 
(col. 4, lines 48-52). 
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6. Claims 18-24 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Langetal. (US. Patent No. 6,314,420) in view of Morag (US. Patent No. 6,324,545). 
Regarding on claims 18 and 24, Lang teaches method comprising: 
retrieving multimedia objects according to a content-based retrieval process; 
presenting the multimedia object to a user; 

monitoring feedback from the user as to which of the multimedia objects are 
relevant; and 

Lang does not explicitly teach annotating one or more of the multimedia objects 
based on the user's feedback. However, Morag teaches, "a user enters an annotation 
into a camera while acquiring the image. This annotation may be a text annotation or it 
may be a voice annotation (col. 9, lines 35-38). This teaches the annotation of the 
image. Therefore, it would have been obvious to one ordinary skill in the art at the time 
of the invention was made to combine the teaching of Morag and Lang because utilizing 
the comments on the images would allow images to be retrieved by other keywords that 
are assign by other users. 

Regarding on claim 19, Morag teaches the monitoring comprises monitoring both 
feature-based relevance feedback and semantic-based relevance feedback (col. 9, lines 
35-40). 
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Regarding on claim 20, Morag teaches the annotating is hidden from the user 
(col. 9, lines 35-40). 

Regarding on claim 21 , Morag teaches the annotating comprise: 

in an event that a particular multimedia object is deemed relevant by the user 

and not yet annotated with the keyword, adding the keyword to the particular multimedia 

object (col. 9, lines 35-40); and 

in an event that the particular multimedia object is deemed relevant by the user 

and is already annotated with the keyword, strengthening an association between the 

keyword and the particular object (col. 9, lines 35-40). 

Regarding on claim 22, Morag teaches the annotating comprises: 

In an event that a particular multimedia object is deemed irrelevant by the user 

and is already annotated with the keyword, weakening an association between the 

keyword and the particular multimedia object (col. 9, lines 35-40). 

Regarding on claim 23, Morag teaches the annotating comprises: 

In an event that a particular multimedia object is deemed irrelevant by the user 

and is already annotated with the keyword, removing the keyword from the particular 

multimedia object (col. 9, lines 35-40). 
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7. Claims 25-31 are rejected under 35 U.S.C. 103(a) as being unpatentable over De 

Bonet (US. Patent No. 5,899,999) in view of Morag (US. Patent No. 6,324,545). 
Regarding on claims 25 and 31 , De Bonet teaches a method comprising: 
maintaining association between keywords and multimedia objects, the 

associations being weight to indicate how relevant the keywords are to the multimedia 

objects (col. 8, lines 53-67); and 

retrieving a set of one or more multimedia objects for presentation to a user (col. 

24, lines 35-40). 

adjusting the weights of the associations based on the user's feedback (col. 4, 
lines 53-57). 

De Bonet does not explicitly teach monitoring feedback from the user as to which 
of the multimedia objects are relevant. However, Morag teaches, "the album may be 
generated based on the thumbnail images, and the customer may comment on these 
images, during the upload of the complete image set" (col. 6, lines 52-55). This teaches 
the feedbacks are given from the user. Therefore, it would have been obvious to one 
ordinary skill in the art at the time of the invention was made to combine the teaching of 
De Bonet and Morag because inputting user to comments for the image would allow the 
images to be retrieved with other keywords. 

Regarding on claim 26, De Bonet teaches the retrieving comprises using content- 
based information retrieval to retrieve the multimedia objects (col. 24, lines 25-26). 
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Regarding on claim 27, De Bonet teaches the retrieving comprises using content- 
based information retrieval and semantic-based information retrieval to retrieve the 
multimedia objects (col. 24, lines 35-40). 

Regarding on claim 28, Morag teaches the monitoring comprises capturing both 
feature-based relevance feedback and semantic-based relevance feedback (col. 6, lines 
52-55). 

Regarding on claim 29, De Bonet teaches the adjusting comprises increasing the 
weights of the association between the keywords and the multimedia objects that are 
deemed relevant by the user (col. 4, lines 53-57). 

Regarding on claim 30, De Bonet teaches adjusting comprises decreasing the 
weights of the association between the keywords and the multimedia objects that are 
deemed irrelevant by the user (col. 4, lines 53-57). 
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